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EXECUTIVE  SUMMARY 


1.  BACKGROUND.  A reliable  system  to  control  the  quality  of  procurement 
instruments  and  to  provide  useful  management  information  regarding  document 
quality  is  needed  within  the  Army  Materiel  Development  and  Readiness  Command 
(DARCOM).  Therefore,  the  Army  Procurement  Research  Office  (APRO)  was  tasked 
to  develop  and  test  a quality  improvement  system  for  procurement  instruments 
that  satisfies  both  requirements.  A previous  APRO  report  (613-1)  presented 
the  initial  system  design;  this  final  report  covers  the  test  findings  and 
resulting  recommendations. 

2.  STUDY  OBJECTIVES.  The  objectives  of  this  study  are  to  develop  and  test 
a system  that  (i)  aids  in  the  improvement  and  control  of  the  quality  of 
DARCOM  procurement  instruments  and  (ii)  provides  useful  management  informa- 
tion compatible  with  the  DARCOM  procurement  management  program. 

3.  STUDY  APPROACH.  The  approach  taken  to  achieve  these  objectives  includes 
the  identification  and  evaluation  of  quality  indicators,  a review  and 
analysis  of  existing  software  quality  control  systems  and  techniques,  inter- 
views with  procurement  personnel  , and  the  synthesis  and  test  of  a system 
that  meets  DARCOM's  requirements. 

4.  CONCLUSIONS  AND  RECOMMENDATIONS.  The  Quality  Improvement  System  for 
Procurement  Instruments  (QISPI)  as  described  in  this  and  the  previous  QISPI 
report  has  considerable  potential  as  a statistical  tool  for  procurement 
management  to  control  the  quality  of  their  procurement  instruments.  Both 
simulation  and  application  tests  have  shown  the  system  structure  and  procedures 
to  be  feasible  in  an  operating  environment.  Consideration_should  be  given  to 
implementing  QISPI,  or  selected  segments  thereof,  at  the  Commodity  Commands. 
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CHAPTER  I 


INTRODUCTION 


A.  BACKGROUND 

Because  the  commodity  commands  within  the  US  Army  Materiel  Development 
and  Readiness  Command  (DARCOM)  are  concerned  about  the  quality  of  their  pro- 
curements, the  Army  Procurement  Research  Office  (APRO)  was  tasked  to  develop 
and  test  a quality  control  system  for  procurement  instruments.  The  system 
was  both  to  aid  in  the  improvement  and  control  of  DARCOM  procurement  instru- 
ment quality  and  to  provide  useful  management  information  compatible  with 
the  DARCOM  procurement  management  program.  An  earlier  APRO  report  titled 
"Quality  Improvement  System  for  Procurement  Instruments"  (QISPI)  presented 
details  of  the  initial  development  effort  of  such  a system.^  This  report  con- 
tinues the  description  of  the  QISPI  development  effort  and  includes  the  system 
test  results  and  implementation  recommendations.  System  modifications  based 
upon  test  findings  are  also  presented. 

A familiarity  with  the  previous  QISPI  report  is  assumed  in  many  areas  of 
this  report.  Therefore,  the  reader  will  find  it  helpful  to  review  613-1  before 
reading  further.  This  is  particularly  necessary  to  understand  what  procurement 
instrument  quality  indicators  are  and  what  deficiencies  are  recorded  against 
these  indicators. 

^Norton,  M.  G.  "Quality  Improvement  System  for  Procurement  Instruments," 
Army  Procurement  Research  Office,  Fort  Lee,  Virginia  23801.  APRO  613-1, 
February  1977. 


The  system  structure  as  previously  designed  is  briefly  reviewed  in  the 
remainder  of  this  chapter.  Chapter  II  describes  the  system  tests  conducted 
at  two  DARCOM  commodity  commands.  A discussion  of  test  findings  and  other 
quality  control  considerations  is  presented  in  Chapter  III.  Chapter  IV  gives 
the  study  conclusions  and  implementation  recommendations. 

B.  REVIEW  OF  QISPI 

1 . Operation  of  QISPI. 

Document  quality  measurement  and  control  begins  at  the  commodity 
command  level  with  a quality  indicator  (QI)  calculation  based  upon  the 
number  and  type  of  instrument  deficiencies  per  page.  These  deficiencies  or 
errors  are  detected  through  formal  board  review.  Standard  control  charts 
for  QI  are  used  to  determine  document  acceptability  and  track  quality  levels 
at  the  commodity  commands.  Corrective  action  is  taken  where  needed  according 
to  the  information  provided  by  QISPI. 

The  QI  values  from  each  commodity  command  are  adjusted  at  the  HQ  DARCOM 
level  to  reflect  review  board  proficiency  and  sampling  percentages  which 
results  in  the  final  performance  indicator  (PI)  calculation  for  all  of  DARCOM. 
This  PI  is  a measure  of  outgoing  instrument  quality. 

2.  Quality  Indicator. 

The  quality  indicator  (QI)  used  in  QISPI  to  gauge  instrument  quality 
is  the  number  and  type  of  deficiencies  detected  during  review  by  a procure- 
ment review  board.  The  exact  expression  for  QI  which  is  calculated  for  each 
instrument  at  the  commodity  command  level  is  given  in  equation  (1). 


2 


(1) 


n 


n 


+ A, 

i=1  i=1 


n 


where : 

QI  is  the  weighted  number  of  deficiencies/page, 

B is  the  number  of  minor  deficiencies, 

A is  the  number  of  major  deficiencies, 

C is  the  weighting  constant, 
n is  the  number  of  instrument  pages. 

Depending  upon  the  actual  deficiency  rates  and  sample  size;  i.e.,  number  of 
document  pages,  the  probability  distribution  function  of  the  QI  is  well  des- 
cribed by  either  the  chi-square  distribution  or  normal  distribution.  Chapter 
III  discusses  which  is  more  appropriate  and  when. 

Once  these  QI  values  are  calculated^  they  are  plotted  on  standard  quality 
control  charts  either  individually  or  as  group  averages.  The  control  charts 
are  then  used  to  determine  instrument  acceptability  and  indicate  quality  trends. 

3.  Performance  Indicator. 

To  account  for  both  instrument  sampling  and  deficiencies  that  go 
undetected  during  instrument  preparation  and  review,  a second  calculation  is 
required  called  the  performance  indicator  (PI).  The  expression  for  PI  is 
given  in  equation  (2). 
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(2) 


PI  = (NLI)(^-  QIL)  + (NSI)(^-  (QIS)(SF)) 

NLI + NSI 

where: 

PI  is  the  performance  indicator  in  deficiencies/page, 

NLI  is  the  number  of  instruments  prepared  and  reviewed  100  percent, 
QIL  is  the  average  QI  of  the  100  percent  reviewed  instruments, 

PF  is  the  proficiency  factor  (0  < PF  1.0), 

NSI  is  the  number  of  instruments  prepared  in  the  group  which  was 
sample  reviewed, 

QIS  is  the  average  QI  of  the  sample  reviewed, 

SF  is  the  sampling  factor  (0  < SF  <_  1.0). 

Whereas  the  QI  value  is  essentially  a measure  of  incoming  quality,  the  PI 
value  becomes  a measure  of  outgoing  quality.  The  PI  values  are  normally 
distributed  by  the  Central  Limit  Theorem  since  average  QI  values  are  used  in 
the  calculations. 

Because  solicitation  and  contract  documents  do  not  appear  to  be 
statistically  similar  enough,  a separation  into  two  groups  was  necessary  for 

application  in  DARCOM.  The  expression  for  this  commodity  command  PI  is  given 
in  equation  (3). 
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PI  = (NLC)(^-  QLC)  + (NSC)(^-  QSC-SFC)  + '(NLS)(^-  QLS)  + (NSS)(^-  QSS-SFS) 

NLC  + NSC  + NLS  + NSS 

(3) 

where : 

PI  is  the  performance  indicator  in  deficiencies/page, 

PFC  is  the  review  board  proficiency  factor  for  contracts, 

PFS  is  the  review  board  proficiency  factor  for  solicitations  for  each 
command 

NLC  is  the  number  of  large  dollar  value  contracts  reviewed, 

QLC  is  the  average  QI  for  the  large  dollar  value  contracts 

NSC  is  the  number  of  small  dollar  value  contracts  prepared  for  award 

QSC  is  the  average  QI  of  the  small  dollar  value  contracts  reviewed 

SFC  is  the  percentage  of  NSC  that  were  reviewed  before  award 

NLS  is  the  number  of  large  dollar  value  solicitations  reviewed 

QLS  is  the  average  QI  for  the  large  dollar  value  solicitations 

NSS  is  the  number  of  small  dollar  value  solicitations  prepared  for 
issuance 

QSS  is  the  average  QI  of  the  small  dollar  value  solicitations  reviewed 

SFS  is  the  percentage  of  NSS  that  were  reviewed  before  issuance. 

This  calculation  is  made  at  the  headquarters  level  from  data  provided  by  the 
commodity  commands  as  part  of  DARCOM's  procurement  management  program. 

Once  the  PI  values  are  calculated,  they  are  weighted  by  the  applicable 
number  of  instruments  for  each  command  and  averaged  for  the  DARCOM  PI 
measurement  using  equation  (4). 
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(4) 


m 

Pip  = Y. 

i=1 
tn 

i=1 

where: 

Pip  is  the  DARCOM  Performance  Indicator 

N.  is  the  total  number  of  instruments  processed  by  the  respective 
commands  during  the  reporting  period;  (i.e.,  NLS+NSC+NLS+NSS) 

m is  the  number  of  commodity  corrmands  reporting. 

The  commodity  command  and  DARCOM  PI  values  are  compared  with  previously 
established  targets  of  that  commodity  command  to  determine  if  the  quality 
levels  are  acceptable. 

The  tests  conducted  of  this  system  at  two  commodity  commands  are  described 
in  Chapter  II. 
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CHAPTER  II 


SYSTEM  TESTS 


The  purpose  of  the  test  applications  was  to  determine  whether  or  not  the 
system  procedures  and  development  assumptions  are  reasonable  considering  the 
operating  environment  at  the  commodity  commands.  The  simulation  tests  des- 
cribed in  the  previous  APRO  report  have  shown  the  QISPI  mathematical  structure 
to  be  feasible..  The  application  tests  described  here  show  the  proposed  QISPI 
procedures  to  be  workable  after  some  minor  modifications. 

A.  TEST  PROCEDURES 

The  same  general  test  plan  was  followed  at  both  commodity  commands. 

Briefly,  each  individual  preparing  a procurement  instrument  exceeding  $200,000 
in  value  checked  the  appropriate  column  on  a Procurement  Quality  Checklist 
(shown  in  Figure  1)  and  placed  the  checklist  in  the  contract  file.  A $200,000 
threshold  was  used  for  test  instruments  instead  of  the  usual  $100,000  review 
threshold  because  of  the  limited  time  and  resources  available  at  both  commands. 
During  instrument  review  a review  board  member  completed  the  remaining  columns 
on  the  checklist  by  recording  both  the  number  of  major  and  minor  errors  de- 
tected and  the  number  of  document  pages.  A slightly  expanded  version  of  the 
error  definitions  used  is  provided  in  Appendix  A.  The  instrument  and  file 
were  returned  to  the  preparer  after  review  for  any  necessary  corrective  actions; 
the  checklist  was  filed  for  APRO's  later  use. 

The  process  at  the  DARCOM  commodity  commands  by  which  the 
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reviewers  conduct  instrument  reviews  can  be  illustrated  by  briefly  ex- 
amining line  9,  for  example,  of  the  checklist  in  Figure  1.  The  type  of 

questions  that  must  be  answered  to  determine  whether  the  instrument  is 
acceptable  regarding  the  negotiation  authority  include  the  following; 

1.  Is  a negotiation  authority  appropriate  for  this  procurement? 

2.  If  not  appropriate,  was  it  incorrectly  included? 

3.  If  appropriate,  is  it  cited  correctly? 

4.  If  appropriate,  is  there  sufficient  justification  in  the  contract 
file  for  its  use? 

If  these  questions  are  not  answered  satisfactorily  in  the  instrument  or 
file,  an  error  is  charged  and  recorded  on  the  checklist  in  either  the 
major  or  minor  column  as  appropriate. 

Performance  data  was  not  plotted  on  control  charts  because  of  the  short 
test  periods  - four  months  at  one  command  and  two  months  at  the  other. 

Also,  no  interaction  was  made  with  HQ  DARCOM  as  described  in  the  previous 
APRO  report  since  the  test  concerned  only  the  commodity  command  operations. 

Although  the  test  procedure  was  basically  the  same  at  both  commands,  the 
environment  at  each  differed  in  some  important  ways.  This  provided  a good 
opportunity  to  judge  the  system  with  minor  variations  of  the  same  procedure. 

The  first  command  operates  with  a permanent,  full-time  review  board. 
Contract  and  solicitation  documents  exceeding  the  established  dollar  threshold 
were  subjected  to  a thorough  and  detailed  review  by  an  individual  board 
member  prior  to  the  board  meeting.  For  test  purposes,  errors  detected  by 
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PROCUREMENT  QUALITY  CHECKLIST 

Annotated  Elements  Are  Correct 

Contr 

Specie 

acting 

alist/ 

er 

1 N/A 

Reviewing 

Official 

Contract/Solicitation  No,  Modi f icat ion/ Amendment  No.  PR/Project  No. 

- Offic 

YES 

NO — - 

(Error  Type  A Nuf 

MJ\J0R  1 MINOR 

SECTION  A 

xxxxxxxXxxxxxxxx) 

WxxxxxxxxXxxxxxxx 

— -»-■  ^ * — Ty^^of  Contract,  Modification,  Solicitation,  Amendment,  S Codes 

xxxxxxxx, 

r 

-Q-T 1.  Modification,  Solicitation,  Amendment  Numbers 

xxxxxxxx: 

3.  Issuinq,  Administrative  and  Payina  Offices  and  Codes 

xxxxxxxx 

4.  Delivery  Discount,  and  Invoice  Information 

xxxxxxxx 

5.  Contractor  Name.  Address  and  Code 

6,  Ship  to/Mark  for  and  Codes 

xxxxxxxx 

_ 7.  Accounting  and  Appropriation  Data,  certified  and  available 

8..  Contract/Modification  Amount,' Prices  and  Justification 

Negotiation  Authority,  Included  and  ApDrooriate 

10.  All  Other  Section  A Elements  e.q.  Sole  Source,  SB  Review.  DO  Ratim.  etc 

SECTION  3 

xxxxxxx 

xxxxxxxx 

ixxxxxxxx 

ixxxxxx 

— , — .U.*  ■ Government  Contract/Solicitation  forms  included  with  appropriate  date  etc 

i 

12.  Contractor's  Representations,  Certifications  and  Statements  comoleted 

] — 

SECTION  C 

Y y Y y y Y Y ^ 

f Y Y V V Y Y Y Y ' 

13.  Instructions,  Conditions  and  Notices  included  and  appropriate 

) 

SECTION  D 

xxxxxxx 

:xxxxxxxx 

iixxxxxxxxixxxxxx 

14.,  Evaluation  f_actorSy  e.g.  Discount,  Gov't  Property,  Transportation,  etc. 

t ' '■  ' 

1 

15.  Factors  Other  Than  Price  and  Relative  Order  of  Importance 

i — 

16.  All  Other  Section  D Elements 

1 

SECTION  E 

xxxxxxx 

(xxxxxxxxj 

kxxxxxxxx 

ixxxxxx 

17.  Line  and  Subline  Item  Nos . /NSN/Part  Nos, 

18.  Nouns  and  Quantities 

19.  A>1 1 Other  Section  E Elements 

SECTION  F 

xxxxxxx 

CXXXXXXXX  ! 

ixxxxxxxx: 

ixxxxxx 

20.  Description  and  Sped fication . Including  Revisions  and  Dates 

21.  Brand  fNime  or  Equal  Statement 

22.  All  Othe»*  Section  F Elements 

SECTION  G 

xxxxxxx 

(XXXXXXXX [ 

.xxxxxxxx 

(XXXXXX 

23.  Preservation,  Packaging,  Packing  and  Marking 

SECTION  H 

xxxxxxx 

(xxxxxxxxk 

[XXXXXXXX 

(XXXXXX 

24.  Delivery  or  Performance  Dates  and  Place  and  Method  of  delivery 

1 

1 

SECTION  I 

xxxxxxx 

ixxxxxxxxt 

ixxxxxxxxJ 

:<xxxxx 

25.  Inspection  and  Acceptance  Requirements 

26.  All  Other  Section  I Elements 

SECTION  J 

xxxxxxx 

:xxxxxxxx|( 

Yxxxxxxxx 

ixxxxxx 

27.  Special  Provisions,  e.g.  Option,  Warranties.  First  Article  etc 

i 

SECTION  K 

xxxxxxxji 

ixxxxxxxxk 

KXXXXXXXXjt 

xxxxxx 

28.  Paying  and  ACO  Instructions  Including  PCO  Information 

29.  All  Other  Section  K Elements 

SECTION  L 

XXXXXXXa 

:xxxxxxxxj( 

(XXXXXXXX^ 

XXXXXX 

30.  Required  General  Provisions 

31.  Adduional  Applicable  General  Provisions 

32.  Appropriate  Alterations  to  General  Provisions 

SECTION  M 

XXXXXXX( 

— 11 

xxxxxxxxk 

(xxxxxxxxk 

XXXXXX 

33.  List  of  All  Documents,  Exhibits,  Attachments 

34.  Forms,  Numbers,  Names,  Dates,  and  Number  of  Pages 

xxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxx,xxxxi: 

xxxxxxxk 

xxxxxx.tx\) 

(xxxxxxxxk 

xxxxxx 

35.  TOTAL  NUMBER  OF  DEFICIENCIES 

xxxxxxxkxxxxxxxxkj  1 

36.  TOTAL  NUMBER  OF  DOCUM.ENT  PAGES 

xxxxx.xxxxxxxx>;x.\N! 

37.  QUALITY  INDICATOR  (QI)  ' 

xxvxxxxkxxxxxxx.ikl 

Contract  Specialist  (Signature  A Date)  Contracting  Officer  (Signature  A Date) 

Reviewing  Official  (Sig.  A Date) 

FIGURE  1.  Procurement  Quality  Checklist 
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legal  and  other  reviewers  both  prior  to  and  during  the  board  meeting  were 
also  recorded  on  the  checklists.  Additionally,  this  command  counted  all 
pages  in  the  instrument  package  except  attachments.  This  count  included 
section  M pages  which  consist  primarily  of  DD  Forms  1423. 

The  second  test  command  operates  with  an  ad  hoc  review  board.  The  board 
chairman  is  the  only  permanent  member;  other  members  are  rotated  from  the 
various  functional  areas.  Instruments  exceeding  the  $200,000  threshold  were 
reviewed  by  the  board  chairman  prior  to  the  board  meeting,  and  detected 
errors  were  recorded.  For  test  purposes,  errors  detected  during  the  board 
meeting  by  other  members  were  also  recorded  on  the  checklist.  Errors  found 
during  the  legal  review  prior  to  the  board  meeting  were  not  recorded  on 
the  checklist.  The  second  command  excluded  the  section  M pages  and  attach- 
ments from  the  total  page  count. 

B.  TEST  RESULTS 

QI  scores  obtained  from  the  test  applications  were  calculated  using 
equation  (1)  with  the  weighting  constant  arbitrarily  set  equal  to  four. 

Frequency  histograms  of  the  resulting  QI  values  are  shown  in  Figures  2 and  3. 

Figure  2 shows  the  QI  scores  for  both  contract  and  modification  documents 
for  Command  A which  uses  a permanent,  full-time  review  group.  Test  data  for 
five  solicitations  is  excluded  from  this  figure  because  contracts  and  solici- 
tations are  not  statistically  similar  enough  to  be  treated  the  same.  The  QI 
scores  for  the  small  sample  of  solicitations  averaged  1.78  weighted  deficiencies 
per  page;  the  number  of  pages  averaged  65.  The  average  major  and  minor  error 
rates  for  Command  A were  .118  and  .147  deficiencies  per  page  respectively. 
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FIGURE  2.  Command  A Frequency  Distribution 
of  QI  Values 
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FIGURE  3.  Command  B Frequency  Distribution 
of  QI  Values 


Figure  3 for  Comnand  B shows  scores  for  contract  documents  only.  Contract 
modifications  were  excluded  because  their  small  number  of  pages  caused  highly 
erratic  fluctuations  in  QI  scores.  Command  B,  which  operates  with  an  ad  hoc 
review  board,  had  average  major  and  minor  error  rates  of  .049  and  .027  defi- 
ciencies per  page  respectively. 

Figures  2 and  3 also  show  the  considerable  difference  in  the  mean  and 
range  values  for  number  of  instrument  pages  that  resulted  from  the  different 
cormiand  approaches  to  page  count. 

The  application  test  data  supports  the  assumptions  made  during  system 
development  regarding  error  rates  and  distributions.  Although  the  test  data 
is  somewhat  limited,  it  shows  that  the  chi-square  and  normal  probability  dis- 
tributions are  reasonable  over  the  expected  range  of  instrument  pages  and 
error  rates.  The  selection  of  the  individual  page  as  the  unit  of  measurement 
oyer  the  individual  document  is  also  appropriate  judging  from  the  test  data. 

In  addition  to  QI  scores  and  error  rates,  the  test  applications  provided 
a better  understanding  of  the  system's  potential  usefulness.  QISPI  appears 
to  be  more  useful  for  management  information  and  control  than  as  an  aid  for 
the  procurement  contracting  officers  (PCOs)  who  are  responsible  for  instrument 
preparation.  Mixed  comments  were  received  from  those  who  completed  the  test 
checklists.  Some  thought  it  was  helpful,  but  others  did  not.  The  possible 
regard  of  QISPI  by  operating  personnel  as  a grading  or  rating  system 
naturally  produced  a negative  feeling  in  some. 

Management  personnel  at  the  test  commands  indicated  that  QISPI  could  be 
a good  management  tool.  The  information  it  provides  would  be  useful  in 
improving  and  controlling  the  quality  level  of  their  procurement  instruments; 
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however,  they  did  express  extreme  concern  about  the  use  of  the  command's 
quality  information  at  HQ  DARCOM  level.  They  felt  that  quality  comparisons 
might  be  made  between  commands  without  a complete  understanding  of  the 
procedural  differences  between  commands.  This  concern  is  warranted  since 
meaningful  comparisons  are  possible  only  if  the  instrument  preparation  and 
review  procedures  are  the  same,  and  they  currently  are  not. 

It  took  less  than  15  minutes  extra  each  during  instrument  preparation 
and  review  to  complete  the  checklist  for  the  test.  Since  the  checklist 
serves  to  organize,  guide  and  record  the  work  already  being  done,  little 
additional  effort  was  required.  Some  duplication  of  effort  during  the  test 
period  was  necessary  because  both  commands  have  their  own  document  and  file 
checklists  and  already  record  some  errors  in  the  form  of  board  minutes.  If 
QISPI  is  implemented  at  the  commands  and  the  checklists  and  error  recording 
procedures  are  standardized,  no  duplication  of  effort  will  exist. 

Chapter  III  discusses  how  these  application  tests  have  impacted  the 
initial  system  design  and  operating  procedures. 
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CHAPTER  III 


DISCUSSION  OF  FINDINGS 


A.  TEST  APPLICATIONS 

A number  of  observations  can  be  made  from  the  test  applications  and 
resulting  test  data.  First,  the  skewness  evident  in  Figures  2 and  3 suggests 
that  the  QI  scores  are  better  described  by  the  chi-square  probability  distri- 
bution for  instruments  with  few  pages  and  low  error  rates.  The  normal 
approximation  adequately  describes  all  other  instrument  QI  values.  The 

general  expressions  for  the  expected  value  and  variance  of  the  QI  distribution 

2 

are  given  in  equations  (5)  and  (6)  respectively. 

EV(QI)  = LI  + C-L2  (5) 

and 

Var  (QI)  = (LI  + C^-L2)  (6) 

where 

LI  is  the  minor  error  rate  in  deficiencies  per  page, 

L2  is  the  major  error  rate  in  deficiencies  per  page, 

C is  the  weighting  constant  and 
N is  the  number  of  document  pages. 

Solomon,  Herbert.  Unpublished  memo  regarding  the  QI  distribution. 
Stanford  University,  Stanford,  California  94305.  15  Oct  76. 
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The  QI  expressed  in  equation  (1)  for  any  document  of  N pages  is  a linear 
combination  of  two  independent  Poisson  variables.  The  distribution  parameters 
given  above  reflect  this. 

The  point  beyond  which  the  normal  distribution  is  a good  approximation 
is  not  firm  but  can  be  estimated  for  specific  error  rates  using  simulation 
and  equations  (5)  and  (6).  Figures  4 and  5 show  the  results  of  simulating 
1000  instruments  with  the  error  rates  obtained  during  the  tests;  i.e.,  major 
and  minor  error  rates  equal  to  .12  and  .15  respectively  for  Command  A and 
.05  and  .03  respectively  for  Command  B.  Normality  is  reached  around  50  pages 
per  document  as  shown  in  Figure  4 for  Command  A error  rates  and  around  90  pages 
per  document  as  shown  in  Figure  5 for  Command  B error  rates.  Chi-square 
goodness  of  fit  tests  for  normality  support  the  normal  approximation  for  these 
points.  Once  the  major  error  rate  falls  below  about  .01  deficiencies  per 
page,  the  chi-square  distribution  is  best  used  for  all  instruments  regardless 
of  number  of  pages.  Control  charts  or  tables  developed  for  operating  per- 
sonnel must  be  structured  accordingly. 

The  standard  quality  control  chart  approach  originally  planned  for 
tracking  quality  and  determining  acceptable  performance  may  need  to  be  modi- 
fied slightly  to  minimize  chart  maintenance  and  tracking  effort.  Control 
limits  for  QI  control  charts  are  dependent  upon  the  variance  of  QI  which  is 
a function  of  the  number  of  pages  in  the  instrument.  The  wide  range  in  number 
of  pages  encountered  during  the  test  prohibits  approximation  with  just  one 
or  two  sets  of  limits  with  any  reasonable  degree  of  accuracy.  Although 
separate  limits  would  not  be  needed  for  each  document  page  number,  a minimum 
of  five  limits  grouped  as  follows  would  be  required  to  keep  the  variance 
approximately  constant  within  each  group. 
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Additionally,  the  chi-square  distribution  applies  to  a substantial  portion 
of  the  procurement  instruments,  and  this  complicates  the  use  of  standard  con- 
trol charts.  Upper  and  lower  control  limits  are  not  as  easily  calculated  for 
the  chi-square  as  for  the  normal  distribution.  And  as  quality  levels  improve 
over  time,  the  error  rates  will  decrease  causing  an  even  greater  skewness 
and  chi-square  applicability. 

This  complication  in  determining  acceptable  performance  is  easily  over- 
come by  developing  tables  of  acceptable  error  values.  Figure  6 shows  a 
sample  page  from  such  a table  generated  by  APRO.  The  table  presents  the 
upper  and  lower  error  values  that  correspond  to  the  upper  and  lower  5%  points 
of  its  distribution  for  given  error  rates  and  page  numbers.  LI  is  the  minor 
error  rate  and  L2  is  the  major  error  rate.  If  the  total  weighted  number  of 
errors  for  an  instrument  falls  outside  the  table  values,  assignable  causes 
of  variation  are  present  and  should  be  investigated. 

Acceptance  tables  identify  out-of-control  points,  but  used  alone  do  not 
provide  the  quality  tracking  feature  that  charts  do.  It  will  be  necessary  to 
either  construct  five  separate  charts  for  each  of  the  four  instrument  groups 
or  combine  charts  by  putting  more  than  one  set  of  upper  and  lower  control 
limits  on  some.  Figures  7 and  8 illustrate  the  two  different  approaches. 
Control  limits  in  Figure  7 are  for  a document  in  one  page  range^ say  61  to 
100  pages.  The  outer  set  of  control  limits  in  Figure  8 is  for  a document  in 
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FIGURE  7.  Control  chart  with  one  set  of  upper  and  lower  control  limits. 


NOTE:  X - In-control  point 

0 - Out-of-control  point 
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FIGURE  8.  Control  chart  with  two  sets  of  upper  and  lower  control  limits. 


NOTE:  Out-of-control  points  are  circled. 


one  range  say. 31  to  60  pages,  and  the  inner  set  is  for  a document  of  a 
higher  range  say  61  to  100  pages.  Out-of-control  points  are  determined 
from  the  tables  and  plotted  with  a different  symbol  such  as  "0"  shown  in 
Figure  7 or  circled  as  shown  in  Figure  8. 

Instead  of  having  five  separate  charts  or  some  charts  with  multiple 
upper  and  lower  control  limits,  one  modified  chart  can  be  used  for  each 
instrument  group  but  the  quality  trends  are  less  clear.  The  only  limit  line 
on  the  chart  is  a warning  line  calculated  for  an  average  document  of  say 
60  pages.  This  single  chart  approach  is  illustrated  in  Figure  9.  Out-of- 
control points  are  determined  from  the  tables  and  plotted  with  a different 
symbol  shown  as  "0"  in  Figure  9.  Quality  trends,  although  not  as  readily 
apparent,  are  still  discernible  by  the  frequency  of  out-of-control  points  and 
clustering  around  the  warning  lines.  Of  course,  other  measures  of  quality 
can  be  easily  calculated  and  also  used  to  track  performance  if  desired. 

Average  QI  values  and  percent  out-of-control  over  a period  of  time  are  two 
examples  of  such. 

The  question  of  whether  the  permanent  or  ad  hoc  review  board  is  more 
effective  was  not  addressed  by  this  test.  Both  have  their  advantages  and 
disadvantages.  The  tests  did  show  that  QISPI  can  function  under  either  approach. 
This  study  group  favors  the  permanent  full-time  approach  though,  primarily 
because  QISPI  needs  a thorough  and  consistent  review  to  be  most  effective, 
and  an  ad  hoc  group  cannot  provide  that  as  well  as  a permanent  group  can. 
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FIGURE  9.  Modified  control  chart  for  contract  QI. 


NOTE:  X - In-control  point 

0 - Out-of-control  point 


The  test  applications  surfaced  an  additional  question  concerning  the 
review  procedure:  where  should  procurement  review  and  the  associated  error 
count  begin  in  the  procurement  process?  One  test  command  included  the  legal 
review  comments  in  their  error  count  since  the  legal  review  came  after  pack- 
age submission  to  the  board.  The  other  test  command  did  not  count  the  legal 
review  comments  except  during  the  board  meeting  since  the  package  received  a 
legal  review  prior  to  submission  to  the  board.  Review  and  associated  error 
counting  should  begin  after  the  package  is  considered  complete  and  ready  for 
public  release  by  the  preparer.  If  legal  or  any  other  "review"  is  considered 
part  of  instrument  preparation  before  submission  to  the  formal  board,  comments 
should  be  excluded  from  the  error  count;  otherwise,  review  comments  should 
be  counted. 

No  concensus  was  obtained  for  the  value  of  the  weighting  constant  for 
major  errors.  Although  the  value  is  somewhat  arbitrary,  it  should  reflect 
the  relative  criticality  of  the  two  error  types  regarding  the  degree  of  the 
potential  cost  or  problem  caused  by  not  correcting  the  errors.  Actually, 
error  criticality  ranges  over  a broad  scale,  and  separation  into  just  two 
groups  was  done  for  simplification  purposes.  To  do  otherwise  would  have 
overly  complicated  the  system.  The  value  of  four  used  during  the  tests  is 
satisfactory  and  need  not  be  changed. 

All  pages  except  attachments  can  be  included  in  the  count  of  instrument 
pages.  As  long  as  these  pages  are  reviewed  by  the  review  boards  and  have 
roughly  the  same  probability  of  error  as  other  instrument  pages,  including 
them  is  acceptable  and  increases  the  applicability  of  the  normal  distribution. 


24 


The  QI  data  obtained  from  the  test  applications  is  adequate  to  establish 
only  tentative  performance  targets  and  for  just  the  two  test  conmands.  More 
realistic  targets  can  and  should  be  determined  by  experience  developed  over 
at  least  a six-month  period  at  each  commodity  cormiand.  Judging  performance 
against  an  arbitrary,  unrealistic  standard  is  self-defeating  and  must  be 
avoided.  This  start-up  period  can  also  be  used  to  establish  the  initial 
review  board  proficiency  factors. 

As  discussed  in  the  previous  QISPI  report,  a thorough  and  consistent 
review  is  essential  if  QISPI  is  to  function  properly.  The  degree  to  which 
this  is  accomplished  is  reflected  in  the  proficiency  factors  included  in 
equations  (2)  and  (3).  Instead  of  having  a DARCOM  organization  conduct 
periodic,  independent  reviews  to  establish  these  values  as  considered  earlier, 
a different  approach  is  offered  here  that  is  more  objective  and  statistically 
valid. ^ 

Using  a controlled  experiment  approach,  certain  procurement  instruments 
are  developed  in  advance  of  board  review  such  that  some  document  pages  have  a 
known  number  of  major  and  minor  errors.  After  the  document  has  been  reviewed, 
the  percentage  of  known  errors  detected  gives  a good  estimate  of  the  review 
board  proficiency  factors.  The  general  expression  for  the  resulting  per- 
formance indicator;  i.e.,  measure  of  outgoing  quality,  is  given  in  equation  (7). 


Gelfand,  A.  E.  Unpublished  memo  regarding  software  measurement  error. 
University  of  Connecticut,  Storrs,  Connecticut  06258.  March  1977. 
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(7) 


PT  - - OT 

■ pt^  pr~ 

where : 

PI  is  the  performance  indicator  in  deficiencies  per  page, 

LI  is  the  minor  error  rate  in  deficiencies  per  page, 

PI  is  the  proficiency  factor  for  minor  errors  (0  < PI  _<  1.0), 

C is  the  weighting  constant, 

L2  is  the  major  error  rate, 

P2  is  the  proficiency  factor  for  major  errors  (0  < P2  _<  1.0),  and 
QI  is  the  quality  indicator  in  deficiencies  per  page. 

Incorporating  these  provisions  in  equation  (3)  provides  for  more  accurate 
estimates  of  commodity  command  performance.  Equation  (8)  gives  the  expanded 
version  of  equation  (3). 


PI  = (NLC)(^  + QLC)  + (NSC)(^  QSC'SFC)  + 


(NLS)(^  + - QLS)  + (NSS)(^  + 


C-L2SS 

P2SS 


QSS-SFS) 


NLC  + NSC  + NLS  + NSS  (8) 

where: 

L1LC,  L1SC,  L1LS  and  L1SS  are  the  respective  minor  error  rates  for 
large  and  small  contracts  and  large  and  small  solicitations; 

P1LC,  PI  SC,  P1LS  and  PISS  are  the  respective  proficiency  factors 
for  minor  errors; 
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L2LC,  L2SC,  L2LS  and  L2SS  are  the  respective  major  error  rates; 

P2LC,  P2SC,  P2LS  and  P2SS  are  the  respective  proficiency  factors  for 

major  errors;  and  all  other  variables  are  as  previously  defined. 

This  expansion  introduces  more  accuracy  into  the  estimate  of  outgoing 
quality  at  the  expense  of  some  additional  calculations  and  possible  review 
board  resistance  since  few  people  welcome  continual  grading. 

B.  OTHER  QUALITY  CONTROL  CONSIDERATIONS 

The  quality  improvement  system  described  thus  far  is  one  approach  to 
improve  and  control  the  quality  of  DARCOM  procurements.  Other  quality 

information  sources  can  and  should  be  used  to  complement  or  supplement  QISPI 

where  useful.  The  indirect  procurement  quality  indicators  discussed  in  the 
previous  QISPI  report  can  be  reviewed  from  time  to  time  for  additional  insight 
into  where  problems  exist  and  where  improvements  can  be  made.  Feedback  from 
the  contract  administration  organizations  will  identify  problem  areas  for 
corrective  action.  Comments  from  reviewing  organizations  such  as  the  Inspector 
General  or  Procurement  Management  Review  may  also  identify  areas  where  changes 
can  be  made  to  improve  procurement  quality.  All  of  these  information  sources 
are  currently  available  to  procurement  managers. 

Another  approach  to  procurement  quality  control  that  is  becoming  more 
readily  available  is  to  prevent  errors  from  occurring  before  the  instrument 
gets  to  the  review  board  by  automating  instrument  generation  as  much  as  possible. 
Systems  such  as  the  Procurement  Automated  Data  and  Document  System  (PADDS)^  being 

5 

Smith,  CPT  L.  G.  Letter,  subject:  Procurement  Automated  Data  and 
Document  System  (PADDS),  describing  PADDS.  US  Army  Missile  Command,  Redstone 
Arsenal,  Alabama.  15  June  1976. 
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developed  at  the  US  Army  Missile  Command,  Redstone  Arsenal,  and  the  Automated 
Contract  Writing  Procedure^  being  demonstrated  at  the  Air  Force  Systems 
Command,  Andrews  AFB,  are  examples  of  this  approach.  These  systems  will  not 
only  minimize  form  errors,  but  also  reduce  the  content  error  rate  through 
comprehensive  data  editing  and  validation. 

A modification  to  QISPI  that  has  potential  benefit  after  implementation 
is  to  reduce  the  review  effort  by  sampling  individual  pages  of  the  instruments. 
Instead  of  reviewing  every  instrument  page  as  is  currently  being  done,  a 
random  sample  of  pages  is  taken  and  thoroughly  reviewed  to  estimate  overall 
document  quality.  Various  statistical  sampling  plans  are  available  to  give 
a desired  level  of  assurance  for  minimum  cost. 

In  summary,  the  test  applications  have  demonstrated  the  feasibility  of 
QISPI  in  an  operating  environment.  A major  question  remaining  is  what  is  the 
most  effective  v/ay  to  implement  it?  Chapter  IV  presents  the  recommended 
approach. 


^Meigs,  CPT  0.  Printout  from  a contract  writing  demonstration.  Air 
Force  Systems  Command,  Andrews  AFB,  Washington,  DC. 
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CHAPTER  IV 


CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  Quality  Improvement  System  for  Procurement  Instruments  (QISPI)  has 
considerable  potential  as  a tool  for  DARCOM  procurement  managers  to  control 
the  quality  of  their  procurement  instruments  and,  in  turn,  their  procurements. 
When  implemented  it  will  provide  the  meaningful  management  information  on 
document  quality  that  is  presently  lacking  in  DARCOM  so  that  corrective 
action  can  be  taken  where  needed.  Instrument  quality  control  and  improvement 
is  at  best  haphazard  without  such  information. 

The  system  tests  have  shown  that  the  mathematical  structure  of  the  QISPI 
is  feasible.  The  chi-square  probability  distribution  function  is  appropriate 
to  describe  the  QI  values  of  instruments  with  few  pages  and  low  error  rates; 
the  normal  distribution  adequately  describes  all  other  instruments.  Applica- 
tion tests  have  shown  the  QISPI  procedures  to  be  adequate  and  useful  in  an 
actual  operating  environment.  The  expected  benefits  of  improved  instrument 
quality  and  management  information  far  outweigh  any  additional  administrative 
effort  required  to  implement  and  maintain  the  QISPI. 

Quality  cannot  be  effectively  reviewed  into  a procurement  instrument. 

The  follow-up  action  that  is  taken  as  a result  of  the  information  provided 
by  a control  system  is  what  contributes  to  procurement  instrument  quality 
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in  the  long  run.  QISPI  offers  DARCOM  procurement  managers  a way  to  capitalize 
on  the  statistical  quality  control  techniques  available  today  to  improve  and 
control  the  quality  of  their  procurements. 

B.  RECOMMENDATIONS 

Consideration  should  be  given  to  implementing  QISPI,  or  selected 
segments  thereof,  at  the  Commodity  Commands.  The  implementation 
should  proceed  slowly  at  first,  building  more  sophistication  as  experience 
and  understanding  are  gained.  The  following  specific  recommendations  are 
offered  as  guidelines  for  an  effective  implementation. 

1.  Restrict  QISPI  implementation  to  the  commodity  command  level  initially. 
Once  a data  base  is  established  at  each  command  and  experience  is  gained, 

the  commands  can  then  forward  meaningful  quality  information  to  HQ  DARCOM  for 
their  use. 

2.  Begin  QISPI  implementation  by  recording  and  tracking  errors  in  just 
those  solicitations  and  contracts  that  are  currently  being  reviewed;  i.e., 
those  instruments  over  $100,000  in  value.  Introduce  sampling  of  lesser  dollar 
value  instruments  once  confidence  is  gained  in  the  system. 

3.  Standardize  the  review  and  quality  control  procedures  at  all  commodity 
commands  by  developing  one  common  checklist  and  one  review  approach,  preferably 
with  a permanent,  full-time  review  board.  This  eventual  standardization  is 
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essential  before  performance  can  realistically  be  compared  among  commands. 

The  procurement  quality  checklist  in  Figure  1 with  the  associated  error 
definitions  in  Appendix  A is  a good  starting  point, 

4.  Establish  and  maintain  r eview  board  proficiency  factors  by  periodically 
incorporating  known  errors  into  selected  instruments.  Of  course,  this  effort 
must  be  conducted  by  an  organization  independent  of  the  board  if  it  is  to 
provide  valid  estimates. 

5.  Use  the  acceptance  tables  [Figure  6)  for  determining  out-of-control 
instruments  and  control  charts  (Figure  7)  for  quality  tracking.  If  less 
control  is  needed  after  experience  is  gained  and  instrument  quality  improveSj 
the  modified  charts  (Figures  8 or  9)  can  be  used.  APRO  will  provide  complete 
sets  of  tables  and  control  limits  for  desired  confidence  levels  upon  request. 


31 


SELECTED  BIBLIOGRAPHY 


1.  Gaffney,  M.  W. , Facenda,  J.,  Flynn,  C. , Topolski,  T.  and  Willis,  W. 

"Air  Force  Procurement  Quality,"  US  Air  Force  Academy,  Colorado 
80840.  June  1976. 

2.  Gelfand,  A.  'E.  Unpublished  memo  regarding  software  error..  Uni  vers i tv  of 

Connecticut,  Storrs,  Connecticutt  06268.  March  1977.  

3.  Meigs,  Dan.  Printout  from  demonstration  of  the  Automated  Contract  Writing 

Procedure.  US  Air  Force  Systems  Command,  Andrews  AFB,  Washington,  DC. 

4.  Norton,  M.  G.  "Quality  Improvement  System  for  Procurement  Instruments," 

Army  Procurement  Research  Office,  Fort  Lee,  Virginia  23801. 

February  1977. 

5.  Smith,  L.  G.  Letter  dated  15  June  1976,  to  APRO  describing  the  Procure- 

ment Automated  Data  and  Documentation  System.  US  Army  Missile 
Command,  Redstone  Arsenal , Alabama  35809. 

6.  Solomon,  Herbert.  Unpublished  memo  regarding  the  QI  distribution.  Stanford 

University,  Stanford,  California  94305.  15  October  1976. 


32 


APPENDIX  A 


DEFINITION  OF  MAJOR/MINOR  ERRORS 

NOTE:  In  order  to  determine  if  certain  line  items  such  as  lA,  8C  and  9C  should 
be  charged  with  an  error,  a thorough  review  of  supporting  documentation 
in  the  contract  file  must  be  made.  For  example,  line  item  8A,  contract 
price,  must  be  adequately  supported  by  a price  negotiation  memorandum 
if  no  error  is  to  be  charged. 

SECTION  A 

T!  Type  of  Contract/Modif icaton/Sol ici tation/Amendment 

A.  Inappropriate  type  of  contract  Major 

B.  Code  Incorrect  Minor 

2.  Contract/Modification/Sol ici tation/Amendment  Number 

A.  Number  omitted  Minor 

B.  Incorrect  number  Minor 

3.  Issuing , Administrative,  Paying  Offices 

A.  Information  omitted  Minor 

B.  Incorrect  information  Minor 

C.  Incorrect  code  Minor 

4.  Del i very, Discount  and  Invoice  Information 

A.  Delivery/FOB  Point  omitted  Major 

B.  Delivery/FOB  Point  incorrect  Major 

C.  Discount  omitted  or  incorrect  Major 

(If  no  discount  offered  by  Contractor,  omission 

of  "none"  or  "N/A"  is  not  an  error) 

D.  Omission  of  place  for  submission  of  invoices  Minor 

5.  Conctractor's  Name  and  Address 

A.  Incorrect  Contractor  Major 

B.  Address  incorrect  Major 

C.  Incorrect  code  Minor 

6.  Ship  To/Mark  For 

A.  Incorrect  destination  or  destination  omitted  Major 

B.  MILSTRIP  Data  not  consistent  with  that  provided  Minor 

to  the  Contract  Specialist  or  omitted 

C.  Mark  for  information  omitted  or  incorrect  Minor 
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7.  Accounting  and  Appropriation  Data 

A.  Adequate  certified  funds  not  available  Major 

B.  Accounting  and  Appropriation  data  cited  incorrectly  Major 

C.  Accounting  and  Appropriation  data  omitted  Major 

8.  Contract/Modification  Amount  (Also  Section  E) 

A.  Contract  price  (unit  and  total,  if  applicable)  omitted  Major 

B.  Amount  cited  not  consistent  with  the  negotiated  Major 

agreement  or  competitive  price 

C.  Insufficient  justification  for  contract  amount  Major 

9.  Negotiation  Authority 

A.  Authority  cited  incorrectly  Minor 

B.  No  negotiation  authority,  if  required  Major 

C.  Negotiation  authority  not  appropriate  or  insufficient  Major 

justification  for  its  use 

10.  Other  Section  A Elements 

A.  If  sole  source,  omission  of  sole  source  justification  or  Major 

NCPSA  and  proper  signatures/approvals 

B.  Omission  of  Small  Business  review  Major 

C.  Omission  of  DO/DX  Rating  Minor 

SECTION  B 

11.  Certifications  and  Representations 

A.  Required  certification  and  Representation  omitted  Minor 

B.  Clause  not  in  the  latest  fomicrequired  by  ASPR  or  Minor 

other  regulations 

C.  Inappropriate  provision  included  Minor 

12.  Contractor's  Submission  of  Certifications  and  Representations 

A.  Certification  and  Representation  not  completed  Minor 

SECTION  C 

13.  Instructions,  Conditions,  Notices 

A.  Omission  of  pertinent  information  Minor 

B.  Inclusion  of  inappropriate  information  Minor 

C.  Ambiguous  information  Minor 

SECTION  D 

14.  Evaluation  Factors 

A.  Pertinent  factors  omitted  Major 

(i.e.  discount.  Government  Production  and  Research 


Property,  F.A.  testing,  transportation) 


15. 

Factors  other  than  price 

A.  Pertinent  factors  omitted 

Major 

B.  Relative  order  of  importance  omitted  or  incorrect 

Major 

C.  Factors  not  consistent  with  Evaluation  Plan 

Major 

D.  Evaluation  weight  set  forth  in  contract 

Minor 

16. 

Other  Section  D Elements 

Major /Mi  nor 

SECTION  E 

17. 

Line  and  Subline  Item  Number 

A.  CLIN/SUBCLIN  (number)  omitted 

Minor 

B.  CLIN/SUBCLIN  (number)  incorrectly  identified 

Minor 

18. 

Nomenclature,  NSN,  Part  Number,  Quantity 

A.  Omission  of  data 

Major 

B.  Incorrect  Nomenclature 

Ma  j or 

C.  Incorrect  NSN  or  Part  Number 

Major 

D.  Incorrect  Quantity 

Major 

19. 

Other  Section  E Elements 

Major/Mi  nor 

SECTION  F 

20. 

Description  and  Specifications  including  Revisions  and  Dates 

A.  Description  or  Spec  omitted 

Major 

B.  Revision/date  omitted  or  unidentified 

Major 

C.  Incorrect  description,  specification,  or  revisions 

Major 

21. 

Brand  Name  or  Equal  Statement 

A.  Statement  omitted 

Major 

B.  Statement  incorrect 

Ma  j or 

22. 

Other  Section  F Elements 

Major/Mi  nor 

SECTION  G 

23. 

Preservation,  Packaging,  Packing,  Marking 

A.  Information  omitted  or  incorrect 

Major 

SECTION  H 

24. 

Delivery  or  Performance 

A.  Information  omitted  or  incorrect 

Major 

B.  Place  and  method  of  delivery,  if  FOB  destination 

Major 

C.  Place  and  method  of  delivery,  if  FOB  origin 

Minor 

SECTION  I 

2^r 

Inspection  and  Acceptance  Requirements 

A.  Inspection  and  Acceptance  point  omitted  or  incorrect 

Major 

A-3 


Major 


26.  Other  Inspection  or  Acceptance  Requirements  (i.e.  Fly-to-Buy) 
omitted  or  incorrect 

SECTION  J 

27.  Special  Provisions 

A.  Option,  EPA,  Incentive  arrangement,  F.A.  approval. 
Warranty  Clause,  GFP  and  approvals,  DTC,  Award  Fee 
ProvisioJis  omitted,  incorrect  or  ambiguous 

B.  Special  Provisions  ambiguous  or  inappropriate  or 
proper  approvals  not  obtained 

C.  Technical  liaison  clause  omitted  or  incorrect 

D.  Other  Special  Provisions 

SECTION  K 

28.  Paying  and  ACO  Instructions  Including  PCO  Information 
A.  Information  omitted  or  incorrect 

29.  Other  Section  K Elements 
SECTION  L 

30.  Required  General  Provisions 

A.  Provision  omitted  or  not  current 

31.  Additional  Applicable  General  Provisions 
A.  Provision  omitted  or  not  current 

32.  Appropriate  Alteration  to  General  Provisions 
A.  Alteration  omitted  or  incorrect 

SECTION  M 

33.  List  of  all  Documents,  Exhibits,  Attachments 

A.  If  document,  exhitaits,  attachments  omitted  but  elsewhere 
i ncorporated 

B.  If  omitted  and  not  incorporated  elsewhere  in  contract 

34.  Form  Number,  Name,  Date,  Number  of  Pages 

A.  Omission  of  information 

B.  Incorrect  information 


Major 

Major 

Minor 

Major/Minor 

Minor 

Minor 

Major 

Major 

Major 

Minor 

Major 

Minor 

Minor 
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APPENDIX  B 


STUDY  TEAM  COMPOSITION 
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Dakota  State  University,  1969;  M.E.  Industrial  Engineering,  Texas  A&M  University, 
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Exchange  (DLSIE).  Before  that,  Mr.  Norton  was  a General  Engineer  with  the 
Safeguard  System  Command,  Alabama  and  has  been  a Government  subcontractor. 

Robert  W.  Nick,  Procurement  Analyst,  US  Army  Procurement  Research 
Office,  US  Army  Logistics  Management  Center,  Fort  Lee,  Virginia;  B.B.A., 
University  of  Mississippi,  1951;  M.S.,  in  Economics,  Ohio  State  University, 

1966.  Prior  to  joining  the  US  Army  Procurement  Research  Office,  Mr.  Nick 
served  as  a member  of  the  Aeronautical  Systems  Division  Procurement  Committee. 

He  also  has  had  experience  as  a contracting  officer,  contract  negotiator  and 
supervisory  purchasing  agent. 
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